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A  Framework  for  Calculating  Indirect  Costs  and  Earned  Value 
for  IT  Infrastructure  Modernization  Programs 

Presenter:  Richard  F.  Suter,  PMP,  is  Principal  Systems  Engineer  at  HPTi  supporting  a 
variety  of  DoD  business  system  modernization  and  acquisition  programs.  Prior  experience 
includes  serving  as  Principal  Enterprise  Architect  and  Project  Manager  with  responsibilities  for 
Process  Improvement,  Change  Management,  Operations  Research,  Computer  Simulation  and 
Modeling. 


Abstract 

Earned  Value  (EV)  supports  proactive  project  management  by  comparing  work 
accomplished  over  time  against  the  cost  and  schedule  of  work  authorized.  This  comparison  is 
essential  to  a  range  of  tasks  such  as  performance-based  acquisition  and  budgeting.  However, 
the  utility  of  EV  as  a  planning  and  management  tool  depends  on  the  accuracy  of  Planned  Value 
(PV)  estimates.  For  Information  Technology  (IT)  infra-structure  modernization  projects,  those 
estimates  are  dominated  by  difficult-to-calculate  indirect  costs — for  the  effort  consumed  in 
communication,  control,  and  coordination  activities.  While  the  DoD  5000  recognizes  and 
recommends  including  indirect  costs  in  Earned-Value  computation,  it  does  not  provide  guidance 
on  how  to  do  so.  However,  a  conceptual  framework  built  around  the  notion  of  communications 
efficiency  can  be  constructed  and  evaluated  using  the  information  resident  in  artifacts  such  as 
Enterprise  Architecture  products,  organizational  capability  and  maturity  assessments,  and 
repositories  of  project  data;  each  of  these  provide  a  basis  for  developing  (parametric)  bounds  on 
indirect  costs  and,  in  some  instances,  direct  estimates.  These  methods  can  be  built  into  an 
Earned-Value  Management  (EVM)  system. 

Key  Terms:  Activity  Base  Costing  (ABC),  Capability  Maturity  Models,  CMMI,  Indirect-Cost 
Estimation,  Infra-structure  Technology  (IT)  Modernization,  Earned  Value,  Enterprise 
Architecture,  Entropy,  Markov  Models,  Perron-Frobenius  Theorem 

Introduction:  The  Problem  Context 

For  knowledge-intensive  enterprises  such  as  complex  COTS  acquisition/integration 
projects,  indirect-cost  estimation  depends  on  the  capability  to  understand,  manage  and  control 
information  dependencies.  Absence  of  that  capability  would  create  unpredictable  consequences 
to  budgets,  schedule,  risk,  and  to  the  performance  of  acquisition  and  IT  infra-structure 
modernization  programs.  Indeed,  inaccurate  estimates  have  long  plagued  these  programs.  For 
example,  KPMG  determined  that  for  48%  of  project  overruns  the  root  cause  was  poor  planning 
and  estimating  (Software  Productivity  Center).  The  Standish  Group  (1995)  found  the  probability 
of  a  software  project  being  cancelled  increased  to  over  50%  as  a  direct  function  of  project 
size, (as  measured  in  function  points).  In  another  survey  of  8,000  projects  in  1995,  the  Group 
found  that  the  average  project  exceeded  its  planned  budget  by  90%  and  its  schedule  by  120% 
(Standish  Group  1995a;  1995b).  In  general,  the  risk  of  failure  for  large  software  projects  is 
significantly  greater  than  for  small  projects  (Humphrey,  2004,  p.  25). 

But,  where  disciplined  processes  are  in  place,  these  risks  are  significantly  reduced  while 
productivity  is  increased.  For  example,  several  recent  studies  of  Team  Software  Process  (TSP) 
showed  significant  gains  in  productivity  due  to  tracking  of  team  and  individual  activity  time  (e.g., 
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for  reviews,  code  development,  meetings,  defect  removal,  etc.).  This  data-driven  tracking 
enabled  the  identification  of  activities  that  added  value  and  of  those  that  did  not  (Team  Software 
Process  (TSP),  2005,  8).  It  also  provides  the  means  to  acquire  indirect-cost  data  that  is 
essential  to  Earned-Value  calculations  and  to  Activity-Based  Costing  (ABC).  However,  this 
“bottom-up”  approach  requires  a  detailed  understanding  of  organizational  activities,  resources, 
and  time  that  may  not  always  be  available.  In  particular,  ABC  estimates  depend  heavily  on 
labor-intensive  data  acquisition  (e.g.,  observing  myriad  activities,  constructing  analyses,  etc.) 
which  limits  the  estimates’  suitability  to  projects  of  limited  duration  (such  as  IT  modernization). 
While  TSP  may  be  suitable  to  software  development  where  the  key  personnel  constitute  a 
limited  group  with  strong  technical  skills,  the  same  cannot  be  said  for  complex  COTS 
applications  that  often  involve  a  wide  range  of  stakeholders. 

Where  a  “bottom-up”  approach  is  not  feasible,  parametric  (e.g.,  statistical)  cost 
estimation  can  be  developed  by  exploiting  artifacts  that  "should  be"  available  in  an  IT  project 
environment — such  as  Enterprise  Architecture  (e.g.,  DoDAF)  products  and  data  from  project 
repositories.  Whatever  the  approach  taken  to  Earned  Value  (EV),  success  depends  on  effective 
management  control,  communications,  and  coordination.  Lacking  that  basis,  EV  estimates 
invariably  will  be  overly  optimistic  as  compared  to  the  actual  costs  incurred  because  there  is 
minimal  capability  to  assess  "true"  project  scope.  Increases  in  project  size,  complexity,  scope, 
volatility,  stringent  performance  requirements  (e.g.,  achieving  minimal  response  latency  in 
networks)  all  amplify  the  potential  for  the  distortion  of  estimates. 

The  factors  influencing  indirect  costs  are  myriad,  and  have  differing  degrees  of  volatility 
and  impact  over  time.  These  factors  include: 

Economies  of  scale,  learning,  capacity  utilization,  system  linkages,  coordination 
and  control,  integration,  timing,  discretionary  policies,  location,  institutional 
factors,  process  maturity  levels,  learning,  geographical  dispersion,  and  team 
experience. 

Various  weighting  schemes  can  be  developed  and  applied  to  these  parameters  for 
various  classes  of  models,  as  will  be  discussed  in  Sections  3  and  4,  below.  Conceptually,  Figure 
1,  below,  illustrates  the  interplay  of  the  factors  driving  the  multi-dimensional  complexity  facing  IT 
project  management. 
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Figure  1.  Program  Technology  Integration  Management 
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2.  Background:  Earned  Value  (EV)  and  Earned-Value  Management  (EVM) 

EVM  is  a  methodology  for  assessing  project  performance  in  terms  of  cost  and  schedule 
variance  over  time.  It  measures  work  actually  accomplished  against  a  schedule  for  contracted 
tasks  at  discrete  points  in  time.  By  systematically  integrating  the  measurement  of  cost, 
schedule,  and  technical  accomplishment  EVM,  promotes  realistic  cost-schedule  estimates 
throughout  the  project’s  development  cycle.  Specifically,  it  integrates  three  data  sources: 

(1 )  Planned  Value  (PV  =  BCWS)  of  work  scheduled  which  defines  what  is  to  be 
accomplished  (funding  authorized).  The  challenge  is  to  identify  and  to  measure  the  indirect 
costs  which  consume  the  vast  majority  of  IT  project  funds. 

(2)  Actual  Cost  of  Work  Scheduled  (ACWS)  which  defines  what  is  spent — that  is, 
whether  anything  was  accomplished,  or  not. 

(3)  Earned  Value  (EV  =  BCWP)  which  measures  what  was  accomplished  within  the  time 
allowed. 


Thus,  work  packages  accepted  as  satisfactory  “earn”  the  cost  of  the  resources 
consumed  to  complete  them.  The  progress  of  a  project  can  be  determined  by  computing: 

►  SV  =  EV  -  PV 

►  CV  =  EV  -  AC 

From  these  basic  relationships,  a  range  of  other  "dashboard"  metrics  can  be  determined 
as  well,  such  as  Estimated  Time  to  Complete  (ETC)  and  Estimate  at  Completion  (EAC). 
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By  determining  EV  at  specific  points  in  time,  a  project  can  be  assessed  against  its 
schedule  to  determine  whether  it  is  “slipping”  or  not.  By  capturing  the  time  value  of  information, 
EV  provides  Decision  Makers  with  the  predictive  capability  that  enables  flexible  response  to 
opportunities  afforded  by  new  technologies,  evolving  conditions,  and  to  joint  collaboration 
requirements — rather  than  simply  reacting  to  them  post  facto.  That  flexibility  strengthens  overall 
program/project/portfolio  integration  and  alignment  with  Agency  mission. 

3.  The  Challenge:  Computing  Indirect  Costs 

The  management  of  information  results  in  the  creation  of  intangible  goods  and  services 
(e.g.,  technical  advice,  activity  coordination,  stakeholder  engagement,  training,  etc.)  that  enable 
a  project  to  converge  to  a  solution,  as  illustrated  in  Figure  1.  The  efficiency  of  information 
management  governs  the  rate  of  convergence  to  a  solution  and  is  influenced  by  the  volatility  of 
factors  such  as: 

►  Stake-holder  preferences 

►  Coordination  of  trade-offs 

►  Timing  of  design  decisions 

►  Schedule  sensitivity 

►  Unforeseen  side  effects  of  decisions 

►  Technical  and  integration  complexity 

►  Architectural  insight 

►  Regulation  and  policy  constraints 

How  quickly  that  volatility  damps-out  depends  on  factors  such  as: 

•  Organizational  process  maturity  (e.g.,  strong  configuration  control) 

•  Ability  to  manage  customer  expectations  (e.g.,  proactive  stakeholder  engagement) 

•  Overall  project-management  capability  (e.g.,  as  measured  by  the  CMMI,  OPM3,  6-Sigma, 
etc.) 

Projects  with  limited  capabilities  and  processes  have  difficulty  managing  this  volatility 

due  to: 


►  Limited  coordination  and  control 

►  Communications  with: 

S  Low  information  content 
V  Limited  accuracy  &  timeliness 
■S  High  distortion  &  error  rates 
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These  factors  are  unlikely  to  improve  over  the  project  life-cycle  without  changes  to 
overall  process  and  management  capability;  this  unlikelihood  is  one  reason  why  Fred  Brooks 
observed  that  “adding  resources  late  to  a  project  makes  it  later"  (1995).  There  is  an  important 
caveat,  however;  small-scale  projects  provide  environments  where  face-to-face  communications 
are  sufficient  for  most  management,  control,  and  coordination  tasks.  These  conditions  make 
Agile  Methods  feasible  for  software  development.  Unfortunately,  these  agile  methods  do  not 
appear  to  scale-up  to  larger  projects  with  complex  integration,  change  control,  program 
management  and  coordination  requirements. 

An  abstract  representation  of  key  explanatory  variables  underlying  communications 
efficiency  can  provide  important  insight  into  Planned-Value  estimation  calculations,  thereby 
contributing  to  the  construction  of  indirect-cost  estimates.  In  the  physical  sciences  and 
Information  Theory,  the  measure  of  communications  efficiency  is  called  "Entropy."  It  is  used,  for 
example,  to  determine  how  many  distinct  symbols  are  required  to  accurately  represent  the 
content  of  a  message.  The  question  before  us,  however,  is  not  the  number  of  distinct  symbols 
required,  but  rather  the  type  and  extent  of  processes  that  must  be  in  place  to  deliver  a  required 
level  of  communications  accuracy  (e.g.,  signal-to-noise  ratio  gain).  In  effect,  the  processes 
providing  these  capabilities  can  be  interpreted  as  “invariants”  in  a  dynamic  system.  (Without 
going  into  detail,  entropy  serves  as  a  measure  of  “invariants”  in  a  number  of  fields,  a  discussion 
of  which  can  be  found  in  Lind,  D.,  Marcus,  B.  (1995).  Symbolic  Dynamics  and  Coding  Theory , 
Boston:  Cambridge  University  Press).  The  process  invariants  of  concern  to  us  are  provided  by 
process  management  and  improvement  methods  such  as  the  CMMI,  OPM3,  6-Sigma,  or  their 
equivalents.  As  Figure  2  indicates,  at  least  one  determinant  of  estimate  variability  for  IT  projects 
is  capability  level — this  determinant  can  be  used  to  define  worst-case  bounds  (or  confidence 
intervals)  on  the  accuracy  of  direct-  and  indirect-cost  estimates. 


Figure  2.  Estimate  Variability  as  a  Function  of  CMMI  Level 

[Insufficient  data  was  available  at  the  time  of  the  study  to  make  valid  determinations 
_ for  CMMI  Levels  4  and  5], _ 
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3. 1.  Earned  Value  Dilemmas — For  Software  Intensive  Systems 

As  Figure  2  suggests,  low  capability  projects  are  least  likely  to  develop  valid  estimates. 
Because  they  typically  have  no  improvement  capabilities,  they  are  unlikely  to  acquire  estimation 
skills.  The  problem  is  compounded  by  the  fact  that  for  IT  modernization  projects,  Planned-Value 
(PV)  estimates  are  most  unreliable  early  in  the  project  development  cycle — when  they  could 
provide  the  greatest  benefit — due  to  factors  such  as  initial  instability  and  uncertainty  concerning 
project  scope,  requirements,  and  complexity,  and  schedule.  At  the  onset  of  the  project 
development  cycle,  these  factors  also  will  render  bottom-up,  and  labor  intensive  methods  of 
indirect  cost  estimation,  such  as  ABC,  which  are  difficult  to  apply  regardless  of  project  maturity 
level.  (The  possible  exception  is  organizations  with  sufficient  capability  and  resources  to 
implement  TSP.  Of  course,  the  programs  resulting  from  such  projects,  once  deployed  of  course, 
could  benefit  greatly  from  the  application  of  methods  such  as  ABC.) 

However,  the  Software  Engineering  Laboratory  (SEL)  at  the  NASA  Goddard  facility  has 
computed  cost-estimate  variability  bounds  for  CMMI  Level-3  projects,  the  estimated  growth  in 
cost  projections,  and  the  accuracy  of  those  projects  as  a  function  of  the  project-development 
cycle.  As  illustrated  in  Figure  3,  their  findings  suggest  that: 

(1)  Organizations  with  at  least  CMMI  level-3capability  can  (reasonably)  project  a  40% 

growth  from  an  initial  (under)  estimate. 

(2)  Projects  with  mature  processes  can  realize  significant  reductions  in  estimate  uncertainty 

over  the  development  cycle — increasing  confidence  in  their  estimates. 

Thus,  even  though  projects  may  start  with  similar  levels  of  size,  complexity,  uncertainty 
and  volatility,  those  with  greater  CMMI  or  similar  capabilities  can  more  effectively  utilize  the 
information  resources  at  their  disposal  to  drive  down  estimate  variability.  Yet,  projects  with  less- 
mature  processes  are  unlikely  to  do  so,  regardless  of  the  methodology  employed. 

Therefore,  whether  (and  to  what  extent)  a  costing  method  can  be  effectively  employed 
depends  on  conditions  such  as: 

►  A  disciplined  process-improvement  capability 

►  Automated  Data  Acquisition 

►  Statistical  Process  Control  (SPC) 

►  A  map  of  information  assets  (typically  provided  by  an  Enterprise  Architecture) 

These  are  characteristics  of  projects  with  mature  process  and  management  capabilities. 
Figure  3,  below,  indicates  that  CMMI  level-3  organizations  improve  their  estimates  over  a 
project  lifecycle  by  managing  their  information  resources  and  communications  to  "learn" 
throughout  the  development  cycle;  in  this  way,  they  systematically  reduce  the  uncertainty 
surrounding  virtually  all  decision  variables,  including  cost  estimates. 


Figure  3.  Cost  Estimate  Uncertainty  Reduction  as  a  Function  of 
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Project  Life  Cycle  and  Process  Maturity  Level 
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Milestone  3  Estimate/ 

NASA  Preliminary  1st  release 

Estimate  at  conceptual  design  Milestone  4 


Figure  3:  CMM  Level  3  Projects  at  NASA  -  Software  Engineering  Laboratory  (SEL) 
(http://www.nasa.goddard.gov) 


4.  Parametric  Modeling  of  Indirect  Costs 

Figures  2  and  3  indicate  that  the  ability  to  reduce  the  uncertainty  envelope  is  a  function 
of  project  communications  efficiency,  which  is  governed  by  the  factors  noted  above,  such  as: 

►  Project  maturity  level,  complexity,  scope,  lifecycle  stage 

►  Geographic  dispersion 

►The  volatility  of  project  scope,  stakeholder  preferences  and  requirements 

►Team  experience 

►  Enterprise  Architecture  scope  and  quality 

Project  information  repositories  (i.e.,SEL  at  NASA,  Goddard)  have  data  available  to 
assess  models  that  purport  to  “explain”  how  these  factors  act  as  to  drive  (down)  changes  to 
uncertainty  levels.  For  example,  these  relationships  can  be  stated  more  abstractly  by  letting  E(t) 
represent  the  level  of  Entropy  as  a  function  of  time.  Then,  these  changes  can  be  expressed 
heuristically  as: 

[1]  dE(t)/dt  =  f  (  CMMI  Level,  EA  quality,  SPC,  6-Sigma  capability,  project 
complexity/scope...) 

[2]  dE(t)/dt  =  a^  +  a2x2  +  a3x3  +....  +  akxk  +  e’ 

[3]  Min  E(t)  =  c-ix-i  +  c2x2+  c3x3  +....  +  ckxk  +  e” 

The  term  ‘e’  represents  measurement  errors. 


,,.lsUMUfll  V  HMU>1 
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For  constant  ak,  Eqn  [2]  indicates  a  constant  rate  of  change  over  the  project  lifecycle 
governed  by  the  capability  factors  in  place.  Thus,  if  the  coefficients  ak  represent  a  probability 
distribution,  then  Eqn  [2]  represents  a  constant  rate  overtime,  but  is  probabilistically  distributed. 

Eqn  [2]  assumes  that  the  minimal  Entropy  level  is  a  linear  function  of  the 
activities/resources  consumed  and  the  associated  costs. 

While  there  can  be  many  reasons  for  questioning  the  assumptions  underlying  [1],  [2],  [3], 
they  nonetheless  represent  a  point  of  departure  for  modeling  these  processes,  and  they  make 
subsequent  analysis  mathematically  tractable.  But,  they  could  be  refined  to  include,  for 
example,  non-linear  relationships.  While  this  inclusion  might  improve  explanatory  power — at 
least  heuristically — it  would  risk  a  decrease  in  mathematical  tractability. 

Tractability  issues  aside,  Eqns  [2]  and  [3]  form  a  linear  optimization  problem  that  can  be 
used  to  compute  cost/schedule  estimates  and  other  dashboard  metrics  which  are  subject  to 
constraints  defining  the  structure  of  the  project  (as  determined  by  architectural  and  workflow 
detail,  resource  availability,  etc.).  Eqns  [2]  and  [3]  have  the  effect  of  directly  integrating 
objectives  and  constraints;  Decision  Makers  could  quantitatively  assess  the  value  of  the 
information  provided  by  disciplined  processes  against  a  range  of  “what-if  scenarios  of  scope, 
cost,  performance,  schedule  and  risk  trade-offs  over  a  project’s  lifecycle  by  utilizing  these 
equations.  Other  possibilities  for  extension  and  refinement  include  introducing  time  as  a 
variable,  which  would  transform  Eqns  [2]  and  [3]  into  a  control-theory  problem. 

These  simple  models  provide  qualitative  insight  into  organizational  processes, 
information  dependencies  and  their  impact  on  overall  estimation  capability.  For  example, 

Figures  2  and  3  qualitatively  suggest  that: 

►  Declining  rates  of  estimate  variability  and  uncertainty  decrease  entropy  for  high  capability 
organizations  (CMMI,  OMP3,  6-Sigma,  etc.).  That  is, 

[2]dE(t)/dt  <0 

►  Constant  or  increasing  estimate  variability  for  less  capable  organizations.  That  is, 

[2]  dE(t)/dt  >_  0 

4. 1.  The  Perron-Frobenius  Theorem 

If  the  transition  matrix  P  is  irreducible,  and  a-periodic,  then  Pk  converges  (element-wise) 
to  a  matrix  in  which  each  column  vector  is  the  unique  stationary  distribution  tt’,  independent  of 
the  initial  distribution  tt°. 

[4]  lim  tt'  =  tt°  *  P 
t  ->oo 

Where: 

Irreducible  means  that  all  elements  of  the  matrix  are  non-negative. 

A-periodic  means  no  looping  among  states  (system  movement  among  states  does  not 
result  in  a  periodic  sequence  such  as:  1-2-5...  8-20  ...  1-2-5  ...). 

Ergodic  means  that  for  an  irreducible  Markov  Chain,  a  limit  exists  and  is  independent  of 
“k”  (some  initial  state  for  the  system). 
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[4’]  lim  pkj  (n)  =  tt,,  n  =  0,  1 ,2,  3,....  (discrete  time  intervals) 

That  is  to  say,  after  a  large  number  of  transitions,  the  probability  of  the  transition  from 
state  “k”  to  state  “j”  is  independent  of  “k”,  where  these  states  could  represent  different  values  of 
a  performance  metric.  In  a  project  context,  the  states  could  represent  the  number  of  open-action 
items,  the  processing  time  required  to  complete  a  transaction,  etc.  Thus,  the  long-run  rate  of 
improvement  ttj  could  be  computed  from  time-series  data,  and  could  be  interpreted  as  the  ‘ak’  of 
Eqn  [2];  this,  in  conjunction  with  Eqn  [3],  could  be  used  to  determine  a  range  of  risk  values 
subject  to  a  set  of  constraints — for  example,  on  scope,  cost,  schedule,  resources,  etc. 

If  the  convergence  process  of  Eqn  [4]  is  Markovian,  the  value  of  the  current  state  of  a 
system  at  time  “n”  is  dependent  only  on  the  state  of  the  system  at  time  “n-1 ,”  and  on  no  prior 
states  “n-2,”  “n-3,”  etc.  Thus,  if  the  variable  of  concern  is  estimate  accuracy,  then  its  value 
(state)  depends  only  on  the  accuracy  of  the  estimate  in  the  preceding  time  period.  Of  course,  by 
the  time  a  limiting  value  is  reached,  a  project  could  be  long  since  completed,  have  incurred 
multiple  changes  in  scope,  etc. 

Nonetheless,  Eqn  [4]  is  significant  because  it  quantitatively  integrates  project  scope, 
complexity  and  other  key  parameters  in  the  matrix  P,  with  the  capabilities  and  controls  available 
to  management  in  the  vector  tt.  Those  capabilities  are  largely  a  function  of  its  maturity/capability 
level,  as  measured  by  standards  such  as  the  CMMI,  OMP3,  6-Sigma,  etc.  Those  capabilities 
are  relatively  stable  overtime,  thus  giving  rise  to  the  (relatively)  constant  rates  for  the  vector  tt. 
The  product  of  the  interactions  of  P  and  tt  can  be  used  to  estimate  a  range  of  factors  of  interest 
to  Decision  Makers — assuming,  of  course,  that  valid  data  is  available. 

The  matrix  P  can  be  interpreted  as  a  matrix  of  probabilities  describing  the  likelihood  of  a 
project  being  in  a  state  defined  by  values  for  project  scope,  complexity,  technical  challenge, 
team  geographic  dispersion,  work  flow  and  sequencing  dependencies,  training  levels,  etc.  In 
principle,  this  information  is  available  from  sources  such  as  Balanced  Score  Cards,  Enterprise 
Architectures,  Work  Breakdown  structures,  Subject  Matter  Expert  opinion,  various  assessment 
and  analyses,  project  data  repositories,  etc. 

Per  Eqn  [4],  the  different  capability  levels  represented  in  tt  will  produce  different  rates  of 
convergence  for  the  bounds  on  the  uncertainty  surrounding  parameters  such  as  the  reliability  of 
cost  or  schedule  estimates.  Per  Eqn  [2],  Figures  2  and  3,  project  with  low  capability  levels  are 
unlikely  to  drive  down  the  level  of  uncertainty,  while  more  mature  projects  will  do  so-  and 
overtime  improve  their  estimation  capabilities  -  in  no  small  part  as  a  consequence  for  their 
underlying  communications  efficiency. 

5.  Change  Effects  and  Indirect  Cost  Estimation 

The  larger  the  amount  of  new  technology  to  be  integrated  and/or  modified,  the  larger  the 
risk  of  schedule  and  cost  creep  due  to  factors  such  as  rework  and  overall  under-estimation  of 
the  complexity.  These  considerations  are  particularly  important  for  estimating  the  level  of  effort 
and  costs  associated  with  the  integration  of,  or  modification  to,  large-scale  legacy  systems  and 
COTS  applications.  The  following  diagram  illustrates  these  considerations. 
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Figure  4:  “As-ls”  Financial  Architecture  of  a  Major  Government  Agency 


Figure  4  illustrates  how  cost  under-estimation  can  occur,  especially  in  organizations  with 
low  communications  efficiency  (e.g.,  the  capability  to  systematically  engage  stakeholders, 
technical  staff,  users,  etc.)  and  minimal  architectural  insight  (e.g.,  the  capability  to  construct  a 
system-connectivity  map  such  as  Figure  4).  It  also  illustrates  why  life-cycle  maintenance  costs 
can  be  far  higher  than  in  the  initial  development  cost.  Without  these  capabilities,  the  risk  of 
unforeseen  and  expensive  side  effects  is  high.  However,  with  improved  capability  levels,  the 
ability  to  acquire  and  to  use  a  range  of  information  sources — and,  thus,  the  ability  to  identify  the 
scope  of  secondary  effects,  the  associated  cost,  schedule,  and  scope  impacts — improves.  The 
information  and  data  acquired  could  be  applied  using  Eqn  [4],  the  output  of  which  could  then  be 
used  to  develop  the  constrained  optimization  formulation  of  Eqns  [2]  and  [3]  for  assessing  risk, 
technical  and  programmatic  performance  resulting  from  investments  in  various  capabilities 
across  a  range  of  “what-if”  planning  scenarios. 


5. 1.  Applications  to  System  Test  Coverage 

These  models  could  also  be  used  to  help  quantify  the  scope  and  cost  of  testing  for  IT 
modernization  programs.  Those  costs  are  always  major  portions  of  an  IT  budget,  and  there  is 
no  satisfactory  means  to  answer  the  question  “how  much  is  enough?”  Without  going  to  detail, 
these  equations  could  be  used  to  estimate  test  coverage  requirements  such  as: 

■  The  number  of  modules  requiring  modifications 
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■  The  complexities  of  the  modules  requiring  modifications 

■  The  availability,  accuracy,  completeness  of  the  specifications  of  the  modules  requiring 
modification 

■  The  degree  to  which  modifications  in  a  module  cause  modifications  in  other  modules 
(i.e. ,  secondary  ripple  effects  that  can  have  significant  cost  or  schedule  impact  as  is 
illustrated  below) 

■  The  impact  to  the  reliability  of  the  System 

For  example,  an  estimate  of  the  impact  of  changes  to  systems  with  (probable) 
dependencies  on  other  systems  comprising  an  application  and  the  Total  Expected  Number  of 
Changes  (T)  can  be  computed  using: 

[5]  T  =  A*(l  -  P)'1 

(A  discussion  of  Eqn  [5]  and  its  relationship  to  the  previous  equations  can  be  found  in 
any  standard  text  on  Stochastic  Processes  or  Operations  Research.) 

I  is  the  identity  matrix 

A  is  a  matrix  of  initially  planned  changes 

P  is  the  matrix  of  probabilities  of  changes  that  will  impact  additional  systems.  These 
changes  will  arise  from  a  range  of  project-specific  structural  factors  as  well  as  technical  and 
programmatic  dependencies  that  can  be  determined  from  sources  such  as  DoDAF  Enterprise 
Architecture  products: 

•  OV-3  Operational-Information  Exchange  Matrix 

•  SV-1  System-Interface  Description 

•  SV-6  System  Data-Exchange  Matrix 

These  artifacts  enable  the  determination  of  the  location  and  structure  of  dependencies 
that  create  high  coupling  (e.g.  lots  of  poorly  documented,  ad  hoc  interfaces)  and  low  cohesion 
(software  functionality  not  organized  for  efficient  utilization  or  maintenance)  which  are  typical  of 
(patchwork)  legacy  systems. 

5.2  Modeling  Change  Propagation  Effects — A  Simple  (Hypothetical)  Example 

•  Make  two  changes  to  module  Ml  and  one  change  to  module  M3. 

•  The  question  is  how  many  secondary  changes  will  be  generated  before  the  ripple  effect 
dies  out. 


Ml 

/  \ 

M2  M3 
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Assumptions: 

•  For  any  change  to  a  module,  there  is  a  10%  chance  of  having  to  make  another  change 
to  the  same  module. 

•  For  a  change  Module  Ml ,  there  is  a  20%  chance  of  having  to  make  changes  to  modules 
M2  and  M3. 

•  Changes  to  M2,  M3  do  not  affect  other  modules. 

Table  1.  Initial  Change  Matrix  for  a  (Hypothetical)  Simple  System 

A  = 

Module  #  of  Changes 

Ml  2 

M2  0 

M3  1 


Table  2.  Probability  Connection  Matrix 
P  = 

Ml  M2  M3 

Ml  .1  .2  .2 

M2  0  .10 

M3  0  0  .1 

Total  Expected  Number  of  Changes  (T) 

[4’]  T  =  A*(l  -  P)'1  =  4.31 

So,  the  3  initial  changes  result  in  somewhat  over  4  changes  being  made  before  the 
ripple  effect  dies  out. 

5.3  -  Modeling  Change  Propagation  Effects — A  More  Complex  Example 

What  would  happen  if  we  applied  the  above  model  to  a  system  such  as  presented  in 
Figure  4? 

As  the  following  Table  illustrates,  the  result  can  be  significant. 
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Table  3.  Change  Probabilities — A  More  Complex  (Hypothetical)  Example 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

1 

0.2 

0.1 

0 

0 

0 

0.1 

0 

0.1 

0 

0.1 

0.1 

0.1 

0 

0 

0 

0.1 

0 

0 

2 

0 

0.2 

0 

0 

0.1 

0.1 

0.1 

0 

0 

0 

0 

0 

0.1 

0.1 

0.1 

0 

0.1 

0 

3 

0 

0 

0.1 

0 

0 

0 

0 

0 

0.1 

0 

0 

0.1 

0 

0 

0 

0 

0 

0 

4 

0 

0.1 

0 

0.2 

0 

0.1 

0.1 

0.1 

0 

0 

0 

0 

0 

0 

0.1 

0 

0.1 

0 

5 

0.1 

0 

0 

0 

0.4 

0.1 

0.1 

0.1 

0 

0 

0 

0 

0 

0 

0 

0 

0.1 

0 

6 

0.1 

0 

0 

0 

0 

0.3 

0.1 

0 

0 

0.1 

0 

0 

0 

0.1 

0 

0.1 

0.1 

0 

7 

0.1 

0 

0 

0.1 

0.2 

0.1 

0.3 

0.1 

0 

0.1 

0 

0 

0 

0.1 

0 

0 

0.1 

0 

8 

0.1 

0.1 

0 

0.1 

0.2 

0 

0.1 

0.4 

0 

0.1 

0 

0 

0 

0.1 

0 

0 

0 

0.1 

9 

0 

0 

0 

0 

0 

0 

0 

0 

0.1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

10 

0.1 

0 

0 

0 

0 

0.1 

0.1 

0.1 

0 

0.4 

0.2 

0.1 

0 

0.1 

0.1 

0.1 

0.1 

0.1 

11 

0.1 

0 

0 

0.1 

0 

0 

0 

0 

0 

0.2 

0.3 

0.1 

0.2 

0 

0 

0 

0 

0 

12 

0.2 

0 

0 

0 

0 

0.1 

0 

0 

0 

0 

0.2 

0.3 

0 

0 

0.1 

0.1 

0 

0.1 

13 

0.1 

0.1 

0 

0 

0 

0.1 

0.1 

0.1 

0 

0.2 

0.1 

0 

0 

0.2 

0 

0 

0 

0 

14 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.3 

0.2 

0 

0 

0 

0 

15 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.2 

0 

0 

0 

16 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.2 

0 

0 

17 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.2 

0 

18 

0 

0 

0 

0 

0.1 

0 

0.1 

0 

0 

0.1 

0 

0 

0 

0 

0 

0 

0 

0.3 

Table  4.  The  Change  Propagation  Impact  in  a  More  Complex  System 

Module  Initial  changeTotal  Change 

#  A  T  =  A*(l  -  P) 1 


1 

2 

242 

2 

8 

101 

3 

8 

4 

5 

28 

249 

6 

12 

230 

7 

8 

229 

8 

28 

257 

9 

4 

4 

10 

8 

319 

11 

40 

239 

12 

12 

131 

13 

16 

128 

14 

12 

157 

15 

296 

2961 

16 

28 

150 

17 

28 

188 

18 

40 

139 

Total 

296 

2961 
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5.4-  Discussion 


Table  3  shows  that  for  a  296  initial  changes,  there  is  a  ten-fold  increase  in  the  total 
number  of  changes  to  the  system  which,  if  not  accounted  for,  will  derail  resource  planning,  cost 
and  schedule  estimates.  The  probabilities  in  the  above  Table  are  conservative.  For  poorly 
documented  legacy  systems,  the  probability  dependencies  could  be  much  more  numerous  and 
more  likely.  Such  effects  can  be  expected  for  poorly  documented/engineered  legacy  systems 
that  accumulate  a  large  number  of  patches  over  the  years  and  which  result  in  a  much  larger- 
than-anticipated  number  of  (costly)  ripple  effects.  Improvements  that  benefit  estimation  also 
benefit  many  other  project  components.  Accordingly,  low  maturity/capability  projects  lack  the 
structural  pre-requisites  for  efficient/timely  communication;  they  cannot  marshal  relevant 
information  and  resources  to  identify  or  to  manage  these  effects.  Thus,  estimates  are  typically 
off  by  orders-of-magnitude. 

6.  Conclusion  and  Next  Steps 

For  large  scale  (DoD)  projects  with  hundreds  of  stakeholders,  hundreds  or  thousands  of 
(sub)  systems,  users  and  stakeholders  deployed  globally  at  a  large  number  of  different  sites,  the 
risks  of  under  estimation  are  correspondingly  greater.  Mitigating  those  risks  depends  on  the 
efficiency  of  project/program  communications,  which  is  closely  related  to  overall  project  maturity 
level. 


The  framework  outlined  here  proposes  to  use  the  equations  such  as  [1]  -  [5]  outlined 
above  to  measure  the  outcomes  of  project-management/organizational  effectiveness  as  a 
consequence  of  the  interactions  between  the  capabilities  provided  at  different  maturity  levels 
and  project  characteristics  such  as  scope  and  complexity.  To  make  these  equations  useful 
estimation  tools,  project  data  repositories  such  as  those  of  NASA/SEL,  NIST,  and  others,  will  be 
investigated  to  develop  “realistic”  parameter  values,  model  relationships,  and  confidence  levels. 

Once  validated,  the  models  can  be  used  to: 

►  Calculate  indirect  costs  for  inclusion  in  EV  calculation,  with  confidence-interval  estimates 
defined  in  terms  of  the  interactions  between  project  complexity  and  capability  levels 

►  Add  the  dimension  of  project  communications  efficiency  to  risk  management 

►  Use  the  models,  in  conjunction  with  “what-if”  scenario-capable  Enterprise  Architecture 
tools  (such  as  Metis)  to  provide  both  qualitative  and  quantitative  insight  into  the  effects  of 
trade-offs,  mitigation  strategies,  and  project-improvement  initiatives 

►  Provide  conceptual  insight  into  the  communications  efficiency  of  various  organizational 
models 

►  Embed  the  models  in  a  larger  Planning  Programming  Budgeting  Execution  (PPBE)  and 
Portfolio  Management  framework  and  apply  to  tasks  such  as  understanding  the  cost, 
scheduling  implications  project/program  alignment  with  Agency  mission,  prioritizing, 
stabilizing,  and  managing  joint  requirements,  stakeholder  preferences,  and  assessing  the 
effectiveness  of  “horizontal  integration”  initiatives. 
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Acquisition  Management 

■  Software  Requirements  for  OA 

■  Managing  Services  Supply  Chain 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

■  Portfolio  Optimization  via  KVA  +  RO 

■  MOSA  Contracting  Implications 

■  Strategy  for  Defense  Acquisition  Research 

■  Spiral  Development 

■  BCA:  Contractor  vs.  Organic  Growth 
Contract  Management 

■  USAF  IT  Commodity  Council 

■  Contractors  in  21st  Century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Navy  Contract  Writing  Guide 

■  Commodity  Sourcing  Strategies 

■  Past  Performance  in  Source  Selection 

■  USMC  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  Model  for  Optimizing  Contingency  Contracting  Planning  and  Execution 
Financial  Management 

■  PPPs  and  Government  Financing 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Capital  Budgeting  for  DoD 

■  Financing  DoD  Budget  via  PPPs 

■  ROI  of  Information  Warfare  Systems 

■  Acquisitions  via  leasing:  MPS  case 

■  Special  Termination  Liability  in  MDAPs 
Logistics  Management 

■  R-TOC  Aegis  Microwave  Power  Tubes 
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■  Privatization-NOSL/NAWCI 

■  Army  LOG  MOD 

-  PBL  (4) 

■  Contractors  Supporting  Military  Operations 

•  RFID  (4) 

■  Strategic  Sourcing 

■  ASDS  Product  Support  Analysis 

■  Analysis  of  LAV  Depot  Maintenance 

■  Diffusion/Variability  on  Vendor  Performance  Evaluation 

■  Optimizing  CIWS  Life  Cycle  Support  (LCS) 

Program  Management 

■  Building  Collaborative  Capacity 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  Aegis  and  SSDS 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
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■  Terminating  Your  Own  Program 

■  Collaborative  IT  Tools  Leveraging  Competence 

A  complete  listing  and  electronic  copies  of  published  research  within  the  Acquisition 
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